In some scenes, the low resolution, coverage and other environmental factors of traffic signs lead to missed and false detections in object detection tasks. Therefore, a traffic sign detection algorithm based on improved attention mechanism was proposed. First of all, in response to the problem of low image resolution due to damage, lighting and other environmental impacts of traffic signs, which leaded to the limited extraction of image feature information by the network, an attention module was added to the backbone network to enhance the key features of the object area. Secondly, the local features between adjacent channels in the feature map had a certain correlation due to the overlap of the receptive fields, a one-dimensional convolution of size k was used to replace the fully connected layer in the channel attention module to aggregate different channel information and reduce the number of additional parameters. Finally, the receptive field module was introduced in the medium- and small-scale feature layers of Path Aggregation Network (PANet) to increase the receptive field of the feature map to fuse the context information of the object area and improve the network’s ability to detect traffic signs. Experimental results on CSUST Chinese Traffic Sign Detection Benchmark (CCTSDB) dataset show that the proposed improved You Only Look Once v4 (YOLOv4) algorithm achieve an average detection speed with a small amount of parameters introduced and the detection speed is not much different from that of the original algorithm. The mean Accuracy Precision (mAP) reached 96.88%, which was increased by 1.48%; compared with the lightweight network YOLOv5s, with the single frame detection speed of 10?ms slower, the mAP of the proposed algorithm is 3.40 percentage points higher than that of YOLOv5s, and the speed reached 40?frame/s, indicating that the algorithm meets the real-time requirements of object detection completely.